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Abstract — The  original  Joint  Directors  of  Laboratories  (JDL)  model 
was  developed  in  the  early  90 ’s,  with  revisits  in  1998,  and  2004. 
Today,  with  new  technologies  of  big  data,  cloud  computing,  and 
machine  analytics,  there  is  an  ever  increasing  need  for  integration  of 
people  and  machines.  The  original  JDL  model  focused  on  the  data 
fusion  (correlation,  filtering,  and  association)  issues,  while  today 
there  is  an  increasing  emphasis  on  an  integrated  approach  to 
information  exploitation  over  sensors,  users,  and  missions  using 
enterprise  architectures,  interoperability  standards,  and  intelligence 
to  the  edge.  Given  these  recent  changes  to  computation  and 
distributed  access,  we  examine  ways  for  extending  the  JDL  model 
from  1998  to  support  exploitation  functions  and  information 
management  for  situation  awareness,  massive  data  analytics  for 
contextual  awareness,  and  domain-specific  needs  for  mission 
awareness. 

Keywords:  JDL  model,  enterprise  architectures,  data  fusion, 
information  management,  contextual  reasoning 

I.  Introduction 

This  paper  is  meant  to  serve  as  a  re-examination  of  the  use  of 
the  Joint  Directors  of  Laboratories  (JDL)  model  [1,  2,  3,  4,  5] 
to  meet  the  needs  for  fusion  processing  that  is  responsive  to 
new  technologies  of  big  data,  cloud  computing,  and  machine 
analytics.  The  inception  of  the  original  JDL  Data  Fusion 
model  aimed  at  providing  a  process  flow  for  sensor  and  data 
fusion  [1],  Subsequent  revisions  were  made  in  1998  [2]  to 
incorporate  new  understanding  of  the  issues  involved  in 
exploiting  information  for  diverse  purposes  such  as  impacts 
versus  threats  and  process  refinement  versus  sensor 
management.  There  also  were  needs  to  clarify  terms,  broaden 
the  model  from  its  initial  focus  on  tactical  military 
applications,  and  apply  to  different  users  [3],  With  the 
establishment  of  the  International  Society  of  Information 
Fusion  (ISIF),  the  notion  of  “information”  became  an 
important  concept  in  IF  designs  and  analysis.  In  2004,  the  JDL 
model  was  revised  as  per  the  issues  discussed  from  the 
working  group  [4,  5].  At  the  same  time,  from  the  working 
group,  there  was  the  proposed  Data  Fusion  Information  Group 
(DFIG)  model,  which  included  suggested  updates  from  the 
constituent  member  discussions  such  as  a  focus  on  user 
functions  [6].  The  focused  application  of  this  paper  deals  with 
information  exploitation  of  distributed  multimedia  content 
which  requires  revisions  of  the  JDL  model  for  machine 
analytics,  information  management,  and  user  contextual 
reasoning  over  enterprise  applications. 


Since  the  1990  book  of  Multisensor  Data  Fusion  [7],  the 
process  flow  for  “fusion”  has  been  about  the  same.  However, 
with  the  popularity  of  the  web  and  the  formation  of  ISIF 
(www.isif.org),  new  developments  have  resulted  from 
information  exploitation  and  “management”  [8].  These 
developments  affect  the  existing  models.  An  accepted 
modeling  distinction  has  been  made  between  high-level 
information  fusion  (HLIF)  and  low-level  information  fusion 
(LLIF)  which  was  first  described  in  [7].  The  low-level 
functional  processes  support  target  classification, 
identification,  and  tracking,  while  high-level  functional 
processes  support  situation,  impact,  and  fusion  process 
assessment.  LLIF  concerns  numerical  data  (e.g.,  target 
locations,  kinematics,  and  attribute  types).  HLIF  concerns 
abstract  symbolic  information  (e.g.,  threat,  intent,  and  goals) 
[9].  Over  the  last  decade,  there  have  been  numerous  panel 
discussions  on  HLIF  that  demonstrate  the  ever  changing 
issues  in  systems-level  information  fusion  [10].  Some  recent 
systems  issues  include  new  methods  for  evidential  reasoning 
[11],  situation  awareness  logic  [12],  enterprise  architectures 
[13],  interface  management  [14],  and  distributed  architectures 
[15].  A  complete  list  of  interesting  modifications  to  the  use  of 
the  JDL  functional  model  can  be  found  in  hundreds  of  papers 
in  the  ISIF  conferences. 

The  focus  of  the  paper  is  aimed  at  those  who  are  building 
information  fusion  systems  that  support  information 
exploitation  (InfoEx).  InfoEx  from  many  sensors  and  users 
distributed  across  a  variety  of  locations  requires  a  revisit  to  the 
JDL  model  for  system-based  applications.  For  a  closed-loop 
sensor  system,  the  tenets  of  the  original  JDL  process  flow  are 
applicable  with  real-world  parameters  (e.g.,  a  Kalman  Filter). 
For  InfoEx  systems,  which  are  envisaged  to  have  multiple 
users,  data  sources,  intelligence  data  collections,  missions,  and 
different  users,  the  JDL  model  can  be  used  for  guidance,  but 
has  to  be  adapted  to  meet  these  needs.  A  large-scale  system  is 
difficult  to  comprehensively  model,  and  thus,  provides  a 
notional  representation  of  the  process  flows  across  parameters 
and  conditions  as  done  in  [4]. 

The  key  goal  of  [4]  was  to  clarify  definition  of  terms  and 
establish  a  clear  and  generalized  basis  for  the  JDL  model.  It 
defined  data  fusion  as  'the  process  of  combining  multiple 
pieces  of  data  to  estimate  the  state  of  some  aspect  of  the 
world.  Partitioning  into  data  fusion  "levels"  was  stated  in 
terms  of  the  types  of  such  aspects:  LO:  features,  LI:  individual 
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entities,  L2:  structures  (e.g.,  relationships  and  situations),  L3: 
scenarios  and  outcomes,  and  L4:  aspects  of  the  system  itself. 
Sensor  and  other  information  management  functions  were 
partitioned  into  a  series  of  analogous  levels.  The  model  is  a 
functional  model  and  therefore  agnostic  as  to  whether 
implementation  of  any  of  these  functions  is  by  people  or 
automated  processes  [2],  Extending  the  JDL  model  to 
encompass  resource  management  with  data  fusion  supports 
functions  of  problem-solving  under  uncertainty.  System 
designers  and  process  managers  may  want  a  tight  integration 
of  diverse  fusion,  data  mining,  adaptive  modeling,  sensing  and 
diverse  resource  management  processes. 

Information  exploitation  considers  the  utility  of  information, 
probability  of  acquiring  such  information  via  candidate  action 
plans  and  costs  of  plans,  all  of  which  are  time-dependent 
(utility  and  cost  are  also  user  variables).  InfoEx  involves 
considerations  of  the  characterization  and  representation  of 
utility  and  cost,  multi-agent  and  multi-user  systems,  and 
adaptive  management  and  adaptive  context  exploitation. 
Steinberg  states  [16,  17]: 

Context  provides  expectations,  constrains  processing,  and  can  infer  or 
refine  desired  information  (“problem  variables”)  on  the  basis  of  other 
available  information  (“context  variables  ”) 

A  key  issue  of  utilizing  context  relates  to  the  original 
distinction  of  low-level  information  fusion  (LLIF)  and  High- 
level  Information  Fusion  (HLIF).  In  the  Data  Fusion 
Information  Group  (DFIG)  model  (see  Fig.  1),  LLIF  (LO-1) 
composes  explicit  object  assessment.  HLIF  (L2-6)  composes 
much  of  the  discussions  and  changes  in  the  last  decade  [6]. 


Figure  1  -  DFIG  Information  Fusion  model  (L  =  Level).  [6] 

In  the  DFIG  model,  the  goal  is  to  utilize  the  Data  Fusion  and 
Resource  Management  (DF/RM)  functions  proposed  in  the 
1998  revisions,  while  highlighting  the  user’s  involvement. 
Humans  are  active  observers  with  the  ability  to  reason  over 
contextual  information  [18].  RM  is  divided  into  fusion  process 
control  (L4),  user  refinement  (L5),  and  platform 
placement/resource  collection  (L6),  and  to  meet  mission 
objectives.  L2  Situation  Assessment  (SA)  includes  structures 
and  relationships  which  are  inferred  from  LI  Object 
Assessment  of  individual  entities  from  LO  Data  Alignment 
(i.e.  registered  features).  Since  unobserved  SA  events  are 
difficult  for  a  computer  to  assess,  user  knowledge  and 
reasoning  are  necessary.  L3  (Impact  Assessment)  includes 
scenarios,  outcomes,  sense-making  of  threats,  courses  of 
action,  optimal  decisions  obtained  using  game  theory  [19], 


estimation  of  intent,  etc.  to  help  refine  the  SA  estimation  and 
information  needs  for  different  actions. 

Motivations'.  The  DFIG  model  suggestions  were  revisits  to  the 
original  JDL  model  during  the  2004  discussions.  For  example, 
the  user  requires  different  tasks  utilizing  various  skills 
(perception),  rules  (tasks)  and  cognition  (knowledge)  [20]. 
Also,  many  fusion  systems  are  designed  to  meet  a  specific 
mission.  Thus,  the  Level  5/6  are  distinct  as  to  represent  the 
different  interactions,  tailor  the  information  fusion  design  to 
different  systems,  and  address  the  advancements  in  the  use  of 
information  fusion  by  the  general  community.  However,  there 
are  practical  limitations  to  the  DFIG  model. 

Limitation  1:  The  LI  (explicit  machine  fusion),  L2/3  (implicit 
machine  fusion)  and  interface  to  human  (L5)  are  not  unique. 
Processes  at  any  of  these  levels  can  include  explicit,  implicit, 
human  or  machine  approaches  [6].  A  better  distinction  than 
explicit/implicit  is  between  (1)  filtering/refinement  which 
converts  an  input  estimate  of  a  random  variable  to  a  more 
accurate  or  less  uncertain  estimate  of  that  variable,  and  (2) 
inference/abstraction  which  estimates  (frequently  latent) 
variables  from  those  available  as  inputs. 

Limitation  2\  From  Fig.  1,  the  choice  to  combine  L2/L3  based 
on  common  functions  needs  to  be  separated  based  on  variables 
extracted  for  exploitation.  Distinguishing  (LI,  L2,  L3)  as  data 
fusion  functions  (blocks)  and  (LO,  L4,  L5,  L6)  as  resource 
management  functions  (interfaces)  does  not  characterize  that 
these  blocks  and  interfaces  represent  fusion  and  control 
exploitation  processes.  For  example,  cognitive  reasoning  (L5) 
to  constrain  control  could  also  be  evidential  reasoning  (L2)  for 
fusion.  Planning,  typical  done  by  humans  through  automation 
is  now  afforded  through  autonomy  with  enterprise  resources. 
Finally,  the  platform  of  sensors  and  ground  stations  should 
encompass  other  enterprise  resources  (e.g.,  World  Wide  Web) 
with  varying  degrees  of  controllability  over  not  just  the 
implied  physical  “real  world”,  but  perhaps  the  “external 
universe  of  discourse”  to  cover  virtual  realities. 

Major  suggestions  since  the  2004  revisions  are  the  focus  on 
information  management,  advanced  visualizations,  data 
mining,  and  mission  focus  with  teams,  priorities,  and 
coordination.  All  of  these  functions  help  to  establish  context 
for  the  HLIF  unsolved  issues  of  fusion  and  resource 
management.  For  example,  RM  can  be  aided  by  information 
management  enterprise  computing  (aspects  of  data  acquisition, 
access,  recall,  and  storage  services). 

The  rest  of  the  paper  is  organized  as  follows.  Sect.  II  discusses 
the  information  management  enterprise  with  system  level  (or 
processing)  context  information.  Sect.  Ill  describes  machine 
analytics,  and  Sect.  IV  discusses  user  refinement  in  contextual 
analysis.  Sect.  V  presents  a  notional  example  of  data 
processing  using  physics-derived  (e.g.  video)  and  human- 
derived  (e.g.  text)  fusion  for  both  a  JDL-type  instantiation  and 
suggestions  for  the  JDL-based  information  exploitation. 
Discussions  and  conclusions  are  presented  in  Sect.  VI. 

II.  Information  Management  Enterprise 

In  this  section,  we  detail  the  enterprise,  information 
management,  and  layered  service  constructs  to  support 
management  and  exploitation  of  information  of  context. 


A.  Information  Fusion  Modeling  in  the  Enterprise 

The  current  trends  in  information  fusion  (IF)  are  data  mining, 
enterprise  architectures,  and  communications  [21].  Different 
mission  applications  require  coordination  over  (1)  data: 
models,  storage/accesses  control,  and  process  and  transport 
flow,  (2)  architectures:  (e.g.  service-oriented  architecture),  and 
(3)  the  enterprise  (e.g.  service  bus,  computing  environment, 
and  the  cloud).  Fig.  2  highlights  the  needs  of  the  user, 
elements  of  data  mining  [22],  and  data  flow  in  the  enterprise. 

Information  exploitation  involves  two  processes  (1) 
integrating  and  managing  the  data,  and  (2)  analyzing  the  data 
through  IF  and  data  mining.  The  JDL  model  deals  only  with 
part  of  data  analysis  as  IF.  Services  and  clouds  can  deal  with 
the  implementation  of  these  two  steps  as  represented  in  Fig.  2. 


Information  Awareness 


Figure  2  -  Information  Fusion  Enterprise  Model  (Adapted  from  [23]) 


Context  is  both  input  and  output  of  data  analysis.  For  example, 
context  (e.g.,  terrain  and  background  traffic)  is  used  to  support 
fusion  (e.g.,  finding  usual  behavior)  and  at  the  same  time  is  the 
output  of  fusion  (e.g.,  terrain  from  fusing  video).  Recently, 
Solano  and  Jernigan  [13]  presented  an  enterprise  architecture 
to  manage  intelligence  products  for  mission  objectives 
highlighting  data  formats  (e.g.,  schemas,  unstructured,  and 
metadata);  data  processes  (e.g.,  access,  ingest,  cleansing, 
profiling,  and  ontology  workflows);  and  database  management 
services  (DBMS).  DB  resources  include  contextual 
information  such  as  terrain  models.  Cloud  technology  can 
serve  as  a  basis  for  access  to  DB  resource  information  but 
requires  access  (e.g.,  service-oriented  architecture  (SOA))  to 
enterprise  services. 


•  Providing  tools  to  assess  infonnation  quality  and  suitability; 

•  Exploiting  producer-provided  characterization  of  information  to 
support  automated  management  and  dissemination  of  information 
[9];  and 

•  Tools  for  goal-driven  search,  discovery  and  exploitation  of 
information  to  meet  dynamic  information  needs. 


Optimal  users’  ability/effectiveness  achieved  by  any  of  these 
means  can  make  applications  less  complicated  and  enables  the 
enterprise  to  be  more  agile  to  adapt  to  changing  requirements 
and  environments. 

There  are  several  best  practices  that  help  achieve  the  goals  of 
information  management.  Organizations  will  greatly  improve 
the  interoperability  and  agility  of  their  future  net-centric 
information  fusion  (and  command  and  control)  systems  by: 

1 .  Adopting  dedicated  infonnation  management  infrastructures  (e.g., 
cloud  computing); 

2.  “Packaging”  information  for  dissemination  and  management, 

3.  Creating  simple,  ubiquitous  services  that  are  independent  of 
operating  system  and  programming  language; 

4.  Using  a  common  syntax  and  semantics  for  common  information 
attributes  such  as  location,  time,  and  subject;  and 

5.  Adopting  interfaces  among  producers,  consumers  and  brokers 
that  are  simple,  effective  and  well-documented 


If  appropriately  employed,  these  best  practices  can  reduce  the 
complexity  of  information  fusion  systems,  allow  for  effective 
control  of  the  information  space,  and  facilitate  more  effective 
sharing  of  information  over  an  enterprise  environment. 


Fig.  3  presents  an  IM  model  which  illustrates  the  extended 
relation  of  the  actors  coordinating  through  the  enterprise  with 
the  various  layers  and  inner  circles  providing  the  protocol  for 
information  service  access  and  dissemination  [9]. 
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B.  Information  Management  (IM)  Model 

The  goal  of  IM  is  to  maximize  the  ability  (effectiveness)  of  a 
user  to  act  upon  information  that  is  available  to,  produced,  or 
consumed  within  the  enterprise.  There  are  several  means  by 
which  this  can  be  accomplished: 

•  Reducing  barriers  to  effective  information  use  by  providing 
notification,  mediation,  access  control,  and  persistence  services; 

•  Providing  an  information  space  wherein  information  is  managed 
directly,  rather  than  delegating  IM  responsibilities  to  applications 
that  produce  and  consume  infomiation; 

•  Focusing  on  consumer  needs  rather  than  producer  preferences  to 
ensure  that  information  can  be  effectively  presented  and  used; 


Figure  3  -  Information  Management  (IM)  Model. 

People  or  autonomous  agents  interact  with  the  managed 
information  enterprise  environment  by  producing  and 
consuming  information  or  by  managing  it.  Various  actors  and 
their  activities/services  within  an  IM  enterprise  surround  the 
IM  model  that  transforms  data  into  information.  Within  the  IM 
model,  there  are  various  services  that  are  needed  to  process  the 
managed  information  objects  (MIOs). 

A  set  of  service  layers  are  defined  that  use  artifacts  to  perform 
specific  services.  An  artifact  is  a  piece  of  information  that  is 
acted  upon  by  a  service  or  that  influences  the  behavior  of  the 
service  (e.g.,  a  policy).  The  service  layers  defined  by  the 


model  are:  Security,  Workflow,  Quality  of  Service  (QoS), 
Transformation,  Brokerage,  and  Maintenance,  as  shown  in 
Table  1.  These  services  are  intelligent  agents  that  utilize  the 
information  space  within  the  architecture,  such  as  cloud 
computing  and  machine  analytics. 


Table  1:  Service  Layers 


Security 

Control  access,  Log  transactions,  Audit  logs,  Negotiate 
security  policy  with  federated  information  spaces, 
Transform  identity  and  Sanitize  content 

Workflow 

Manage  workflow  model  configurations,  Instantiate  and 
maintain  workflows,  Assess  and  optimize  workflow 
performance 

QoS 

Respond  to  client  context,  Allocate  resources  to  clients,  QoS 
policy  mediation,  Prioritize  results,  and  Replicate 
information 

Transfer 

mation 

Contextualize  information,  Transform  MIOs,  Support  state 
and  context-sensitive  processing,  Support  user  defined 
processing  functions,  Support  manager  defined  processing 
functions, 

Broker 

Process  queries,  Support  browsing,  Maintain  subscriptions, 
Notify  consumers,  Process  requests  for  information  and 
advertisements,  Support  federated  information  space  proxies 

Maintena 

nee 

Post  MIOs,  Verify  Adherence  to  standards,  Manage  MIO 
lifecycle,  Manage  information  space  performance,  Retrieve 
specific  MIOs  from  repositories,  Support  configuration 
management  of  information  models 

One  recent  technology  development  is  cloud  computing  which 
supports  an  enterprise  analysis  [23,  24]. 


C.  Layered  View  of  the  Cloud 

Using  elements  of  the  JDL/DFIG,  the  enterprise,  and  the  IM 
model  for  sensing,  networking,  and  reporting  can  be  realized. 
Fig.  4  presents  the  layered  information  where  the  end-user 
(operator  or  machine)  desires  quality  information  as  fused 
products  from  data  which  requires  various  methods  and 
services  from  sensor  collections  to  information  delivery. 
“Sensors/Sources”  can  be  viewed  as  a  general  term  as  it  relates 
to  physical  sensors,  humans,  and  database  services  (e.g.,  data 
mining)  that  seek  data  from  the  environment. 


Current  trends  in  information  fusion  share  common 
developments  with  cloud  computing  such  as  agent-based 
network  service  architectures  [25],  ontologies  [26]  and  metrics 
[27]  to  combine  physics-derived  sensing  and  human-derived 
reporting  using  fusion  products. 
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Figure  4  -  Layered  Information  Services. (adapted  from  Kessler,  White  2008) 

D.  Systems-Level  Management  of  Context 

The  JDL  model  is  a  functional  model  that  seeks  to  identify  and 
organize  mathematical  fusion  functions;  with  management 


functions  evident  in  all  levels  of  the  model.  Three  issues  that 
have  been  addressed  as  inherent  at  each  level  are: 

•  Uncertainty  management  through  contextual  awareness, 
and  particularly,  management  of  second-order  uncertainly.  This 
involves  issues  of  source  characterization  (particularly  with 
human  sources),  representation  of  diverse  flavors  of  uncertainty, 
as  well  as  computational  issues  (e.g.  in  PHD-based  methods) 
[28,  29,  30], 

•  Adaptive  context  exploitation,  considering  the  utility  of 
information,  probability  of  acquiring  such  information  via 
candidate  action  plans  and  costs  of  plans,  all  of  which  are  time- 
variable  (utility  and  cost  are  also  user-variable).  This  involves 
considerations  analogous  to  the  above:  consistent 
characterization  and  representation  of  utility  and  cost. 
Additional  factors  include  multi-agent  and  multi-user  systems. 
Aspects  of  this  problem  are  the  integration  of  fusion,  mining  and 
general  problem  solving  methods,  adaptive  process,  model,  and 
goal  management,  and  adaptive  context  exploitation  [31,  32], 

•  Methods  of  user  involvement  through  contextualization,  to 
include  the  interface  and  control  issues  for  specific  tasks  that  are 
operationally  divided  amongst  different  users  such  as 
information  collection,  real-time  control,  forensic  analysis,  and 
visualizations.  These  functions  are  local  functions  distributed 
over  a  large  team  for  specific  tasks  versus  a  single  user  that  is 
responsible  for  the  entire  information  fusion  system[33,  34,  35], 

The  JDL  variations  of  the  Signals,  Feature,  Decision  (SFD) 
Model  [7],  (DF/RM)  dual-node  architecture  [2],  the  User- 
Fusion  Model  (UFM)  [36],  Transformation  of  Requirements 
for  Information  Process  (TRIP)  model  [37],  and  State- 
Transition  Data  Fusion  (STDF)  [9]  each  seek  common 
processes  across  many  levels  and  a  more  robust  way  of 
implementation  by  re-use  of  data  and  methods  to  serve 
different  purposes.  The  method  of  context  has  also  been  a 
subject  of  these  model  developments  and  variations  that  have 
influenced  developers  using  the  JDL  as  a  framework  for 
implementation.  For  the  DF/RM  model,  context  is  divided  into 
those  elements  that  support  data  fusion  (L0-3),  and  those  for 
management  (L4).  An  example  is  terrain  information  in  which 
tracking  can  be  adapted  to  road  information  and  sensors  can  be 
managed  as  related  to  zones  of  authorized  operation.  The 
TRIP  model  divides  the  problem  into  demand  conditioning 
(L4-6)  and  supply  conditioning  (L0-3).  The  demand/supply 
duality  is  meant  to  serve  users  requesting  information  and  the 
system  determining  if  these  requests  can  be  answered.  For 
demand  conditioning,  decision,  exploitation,  and  observability 
context  are  used  to  decompose  the  user  request  into  machine 
functions  (i.e.,  fusion)  that  can  be  processed.  The  UFM  seeks 
to  utilize  the  users  cognitive  requests  (L5-6)  in  establishing 
context  in  their  coordination  of  the  fusion  levels  (LO-4). 
Finally,  the  STDF  utilizes  the  same  control  (L4-6)  to  look  over 
different  functions  such  as  physics-derived  (e.g.  radar)  (L0-1), 
human-derived  (e.g.  text)  (L0-1),  situations,  (L3),  and 
scenarios  (L4).  The  key  attributes  of  these  JDL  variants  are 
inherent  in  establishing  different  methods  of  fusion  and 
control  with  common  issues  described  at  each  level. 

The  future  of  the  use  of  JDL  will  require  developments  in 
uncertainty  analysis,  exploitation,  and  user  involvement  based 
on  the  context,  such  as  mission  management  (L6).  While  these 
related  models  (SDF,  UFM,  TRIP,  DF/RM,  STDF)  all  have 
elements  of  context,  they  also  try  to  solve  real  problems  for 
real  users.  Hence,  a  current  trend  in  information  processing  is 


machine  analytics  that  seeks  methods  of  data  compression  and 
presentation  for  user  interaction. 

III.  Machine  Analytics 

In  the  JDL  model  revisits  in  2004,  little  attention  was  paid  the 
enormous  amount  of  types  of  data  (e.g.  email  and  sensor), 
distributed  locations,  and  various  connections  to  different 
applications  (e.g.  finance  to  surveillance)  that  have  resulted 
from  the  expansion  of  the  World-Wide  Web.  Related  concepts 
recently  emerging  are  machine,  descriptive,  prescriptive, 
predictive,  visual,  and  other  analytics  yet  to  be  coined.  There 
are  three  issues  of  importance  hardware  (e.g.,  Apache  Hadoop 
data  intensive  distributed  architecture),  software  (e.g.,  machine 
analytics),  and  user/domain  applications  (e.g.  visual  analytics, 
text  analytics).  Since  the  JDL  is  a  functional  model,  we  will 
focus  on  the  last  two.  Data  Mining  (DM)  and  data  fusion  (DF) 
are  analysis  functions  supported  by  machines. 

A.  Modeling  Analytics 

The  commercial  business  industry  has  been  managing  large 
volumes  of  transactional  data  using  powerful  databases  systems 
and,  most  recently,  using  cloud  infrastructures.  Traditional 
business  analytics  has  focused  mostly  on  descriptive  analyses 
of  structured  historical  data  using  statistical  techniques.  The 
current  trend  in  DM  is  towards  predictive  analytics  of 
unstructured  data  such  as  documents,  video,  image  sets, 
multimedia  data,  network  data,  matrices,  tensors,  and  graphs 
and  tensors  [38].  The  field  of  data  fusion  [39]  has  been  making 
use  of  cutting  edge  Artificial  Intelligence  (AI)  and  Machine 
Learning  (ML)  techniques  to  perform  situation  and  threat 
assessments.  Given  that  analytics  and  data  fusion  are  two  sides 
of  the  same  coin,  an  effective  revision  of  the  JDL  model 
warrants  a  close  cooperation  between  the  two  communities  in 
terms  of  technology  enrichment  for  managing  and  intelligent 
processing  of  large  volumes  of  data.  The  model  extensions 
should  be  able  to  seamlessly  exploit  such  enriched  technologies 
in  areas  such  as  massive  data  analytics,  hybrid  reasoning,  text 
analytics,  and  distributed  processing. 

Traditional  statistical  approaches  are  invaluable  in  data-rich 
environments,  but  there  are  areas  where  AI  and  ML  approaches 
provide  better  analyses,  especially  where  there  is  an  abundance 
of  subjective  knowledge.  Benefits  of  such  augmentation 
include  mixing  of  numerical  and  categorical  variables  in 
algorithms,  “what-if  or  explanation-based  reasoning, 
explainable  results  of  inferences  easily  understood  by  human 
analysts,  and  efficiency  enhancement  incorporating  knowledge 
from  domain  experts  as  heuristics  to  deal  with  the  “curse  of 
dimensionality”.  Though  early  AI  reasoning  was  primarily 
symbolic  in  nature  (i.e.,  manipulation  of  linguistics  symbols 
with  well-defined  semantics),  it  has  moved  towards  a  hybrid  of 
symbolic  and  numerical,  and  therefore  one  is  expected  to  find 
probabilistic  and  statistical  foundations  in  many  AI  approaches. 
Conversely,  business  analytics  traditionally  has  powerful 
customer  or  other  related  segmentation  techniques  via  various 
powerful  clustering  algorithms  such  k-means,  hierarchical 
clustering  and  k-  nearest  neighbor.  Table  2  depicts  some  of  the 
well-known  techniques  categorized  along  the  statistics,  AI  and 
ML  paradigms  along  with  a  special  category  for  temporal 
reasoning  given  the  dynamic  nature  of  analytics  and  fusion 


problems.  Here  are  some  examples  of  augmentation  and 
enrichment  of  business  analytics: 

•  Enrich  principal  component  and  factor  analyses  with  subspace 

methods  (e.g.,  latent  semantic  analyses); 

•  Meld  regression  analyses  with  probabilistic  graphical  modeling; 

•  Extend  autoregression  and  survival  analysis  techniques  with 

Kalman  filter  and  dynamic  Bayesian  networks,  embed  decision 
trees  within  influence  diagrams;  and 

•  Augment  “nearest-neighbor”  and  £-means  clustering  techniques 

with  support  vector  machines  and  neural  networks. 


Table  2.  Approaches  to  Modeling  Analytics 


Paradigm 

Approach 

Technologies 

Statistical 

Non-deterministic  relationships 
between  variables  are  captured  in 
the  form  of  mathematical  equations 
and  probability  distributions 

Test  hypothesis,  regression 
analyses,  probability  theory, 
sampling,  inferencing 

Artificial 

Intelligence 

Domain  experts  provide  knowledge 
of  system  behavior,  and  knowledge 
engineers  develop  computational 
models  using  an  underlying 
ontology 

Logic-based  expert  systems, 
fuzzy  logic,  Bayesian 
networks 

Temporal 

Linear/nonlinear  equations  specify 
behavior  of  stochastic  processes  or 
of  dynamic  systems  as  state 
transitions  and  observations 

Autoregression,  survival 
analysis,  Kalman  filters, 

Hidden  Markov  Models, 
Dynamic  Bayesian  Networks 

Machine 

Learning 

System  input/output  behavior  is 
observed,  and  M  techniques  extract 
system  behavior  models 

Clustering,  neural  network, 
and  various  linear,  nonlinear, 
and  symbolic  approaches  to 
learning 

Machine  analytics  (MA)  covers  the  broad  spectrum  of 
applications  and  provides  a  direct  link  to  the  traditional  JDL 
model.  Processes  that  require  machine  data  analysis  include: 
physics-derived  sensor  (e.g.,  video),  human-derived  (e.g., 
text),  and  machine  (e.g.,  web  files)  data.  MA  is  also  based  on 
the  processes  of  man-machine  and  machine-machine 
interactions.  Inside  MA  are  the  emerging  concepts  of  (1) 
descriptive,  prescriptive,  predictive  analytics,  and  (2) 
scientific,  information,  and  visual  analytics.  These  concepts 
mirror  discussions  in  the  current  JDL  model  revisits  of 
information  exploitation. 

The  business  oriented  definitions  of  analytics  complement 
elements  of  data  fusion. 

•  Descriptive  Analytics  looks  at  an  organization’s  historical  and 

current  performances  which  can  be  used  to  diagnose  the 
situation. 

•  Predictive  Analytics  forecasts  future  trends,  behavior  and  events 

for  decision  support  such  as  to  suggest  courses  of  action  profile 
and  trending. 

•  Prescriptive  Analytics  determines  alternative  courses  of  actions  or 

decisions  options  (using  predictive  information),  given  the 
historical,  current  and  projected  situations  and  a  set  of 
objectives,  requirements,  and  constraints  [40], 

Visual  analytics  (VA)  [41]  seeks  scientific,  information,  and 
cognitive  representations.  Visualization  supports  analytical 
reasoning,  planning,  and  decision  making  through  effective 
data  representations  and  transformations  over  physical-  and 
human-derived  data  (sometimes  referred  to  hard  and  soft  data 
fusion).  Finally,  user  interaction  with  machines  is  important 
for  the  collection,  exploitation,  and  dissemination  of  data. 

•  Scientific  visualization  deals  with  data  that  has  a  natural 

geometric  structure  (e.g.,  MRI  data,  wind  flows). 


•  Information  visualization  handles  abstract  data  structures  such  as 

trees  or  graphs. 

•  Visual  analytics  is  especially  concerned  with  sensemaking  and 

reasoning. 

If  we  look  at  “analytics”  (Table  3)  it  mirrors  the  JDL  (and 
other  proposed  models  -  DFRM  and  DFIG)  in  having  both  the 
data  fusion  reasoning  (e.g.,  Bayes)  and  systems-level 
management  (e.g.,  control)  functions.  Thus,  the  MA  is  like 
reasoning,  while  VA  is  about  management. 


Table  3.  Machine  Analytics  Mapped  to  Information  Fusion  Levels 


Fusion 

Machine 

Concept 

Level  0 

Scientific 

Access  to  data  and  pedigree  of  information 
and  issues  of  structured/unstructured  data 

Level  1 

Information 
(Visual,  text) 

Development  of  graphical  methods  for  data 
analysis 

Level  2 

Descriptive 

Uses  data  mining  to  estimate  the  current 
state  (i.e.  Machine  learning)  over  different 
reasoning  of  trends  for  modeling 

Level  3 

Predictive 

Future  options  from  current  estimates 

Level  4 

Prescriptive 

Sequencing  of  selected  actions 

Level  5 

Visual 

Sensing  Making  and  Reasoning 

Level  6 

Activity- 

Based 

Policy  instantiation  of  desired  outcomes  as 
to  a  focused  mission 

B.  Data  Fusion  Analytics 

Descriptive  and  Predictive  Analytics  together  establish  current 
and  projected  situations  of  an  organization,  but  do  not 
recommend  actions.  An  obvious  next  step  is  Prescriptive 
Analytics,  which  is  a  process  to  determine  alternative  courses 
of  actions  or  decision  options,  given  the  situation  along  with  a 
set  of  objectives,  requirements,  and  constraints.  Automation  of 
decision-making  of  routine  tasks  is  ubiquitous,  but  subjective 
processes  within  organizations  are  still  used  for  complex 
decision-making.  This  current  use  of  subjectivity  should  not 
prohibit  the  fusion  and  analytics  community  from  pursuing  a 
computational  approach  to  the  generation  of  decision  options 
by  accounting  for  various  non-quantifiable  subjective  factors 
together  with  numerical  data.  The  analytics-generated  options 
can  then  be  presented,  along  with  appropriate  explanations  and 
backing,  to  the  decision-makers  of  the  organization. 

Systems  routinely  collect  and  store  large  volumes  of  data  on  a 
continuous  basis  from  a  variety  of  disparate  and  heterogeneous 
sources.  Though  such  distribution  is  coherent  with  recent 
thrusts  towards  net-centric  warfare,  analysts  often  face  a 
daunting  task  when  searching  for  specific  data  or  for  series  of 
correlated  data  residing  in  distributed  sources.  One  solution  is 
to  build  a  large  centralized  data  storage  area  in  advance,  such 
as  a  cloud  infrastructure.  However,  the  proprietary  nature  of 
some  of  the  sources  requires  that  they  operate  autonomously 
and  hence  a  distributed  fusion  approach  [42]  is  vital. 

There  are  also  other  types  of  analytics:  (1)  Web  Analytics : 
internet  usage  data  for  purposes  of  understanding  and 
optimizing  web  usage,  and  business  and  market  research;  (2) 
Image  Analytics :  real-world  videos  and  images  to  extract 
information  with  machine  performance  comparable  to 
humans;  and  (3)  Cross-lingual  Text  Analytics:  Analytics  with 
contents  in  multiple  languages  that  enable  a  system  to  discover 
and  maximize  the  value  of  information  within  large  quantities 
of  text  (open-source  or  internal).  The  growing  trend  in 


analytics  serves  at  DF/DM  for  both  autonomy  (machine)  and 
automation  (machine  to  user). 

IV.  User  Involvement 

The  Observe-Orient-Decide-Act  (OODA)  model  serves  as 
another  common  widely  used  model,  and  referenced  in  the 
information  fusion  community.  As  differentiated  from  the  IM 
Model  which  deals  with  the  enterprise  and  distribution  of  the 
data,  the  OODA  is  focused  on  a  local  control  loop.  Both  of 
these  models  are  useful  for  information  fusion  and  can  be 
enhanced  with  InfoEx  capabilities  such  as  visual  analytics. 

The  ODDA  model  serves  as  a  good  basis  for  an  operator;  but 
updates  have  appeared  with  multiple  OODA  loops:  TECK- 
OODA  [43],  C-OODA  [44],  and  (Us  versus  Them)  [45],  For 
example  in  Fig.  5,  a  real-time  operator  is  reactive;  whereas  an 
analyst  has  more  time  for  data  mining  and  discovery. 


Figure  5  -  Multiple  OODA  Loop  Human/Machine  Interactions. 

Using  multiple  OODA  loops  [46,  47],  there  are  two  types  of 
cases  where  an  operator  is  making  real-time  decisions  and  an 
analyst  is  making  forensics  non-real  time  assessments.  The 
motivation  is  based  on  the  paradigms  of  (A)  getting  inside  the 
enemy’s  control  loop  (L3),  and  (B)  team  decision  making 
(L5).  Obviously,  the  selection  of  the  competitor,  such  as  in 
game  theory  [19],  requires  determining  a  focused  threat  (L6). 
What  is  extended  from  the  JDL  and  variations  [48]  is 
highlighting  real-time  versus  the  non-real  time  data  mining 
and  machine  analytics  for  adaptive  information  exploitation. 

V.  Analysis  of  Informaton  Exploitation 

With  the  JDL  and  advances  in  computation,  there  is  a  need  to 
focus  on  the  HLIF  management  functions  such  as  managing 
the  amount  of  information  available.  Current  developments  in 
enterprise  architectures  support  information  management, 
cloud  computing,  user  refinement  through  queries,  and 
machine  analytics.  For  contextual  reasoning,  much  of  the 
ancillary  data  is  available  on  a  distributed  enterprise  system  to 
provide  situational  analysis  focused  on  mission  needs.  One 
example  is  that  a  user  would  like  to  refine  the  estimates  of 
many  target  tracks  and  uses  semantic  queries  (ontology)  to 
access  the  database  through  machine  analytics  /data  mining. 

A.  Context  Tracking  with  Physics  &  Human-derived  data 

We  designate  two  types  of  data:  video  and  user  text  (e.g., 
documents).  The  physics-derived  video  can  be  processed  by  a 
machine;  however,  the  human-derived  text  can  be  processed 
by  a  machine,  but  still  needs  user  involvement  to  understand 
the  meaning  of  the  connected  words. 

Information  fusion  developments  include  large  data  (e.g., 
imagery),  flexible  autonomy  (e.g.,  from  moving  airborne 
platforms  over  communication  systems),  and  human 


coordination  for  situation  awareness  [49].  Fig.  6  demonstrates 
an  imagery  data  collection  example  using  electro-optical  (EO) 
cameras  and  Wide-Area  Motion  Imagery  (WAMI).  Using 
information  fusion  for  situation  awareness  based  on  imagery 
includes:  (i)  tracking  targets  in  images  (fusion  over  time)  [50], 
(ii)  identifying  targets  using  different  sensors  (fusion  over 
frequency)  [51],  and  (iii)  linking  target  measurements  over 
wide  areas  (fusion  over  space)  [52].  Enterprise  information 
includes  stored  data  of  the  physical  (terrain),  resource 
(sensors),  and  social  context  (objects)  that  is  easily  accessible 
from  cloud  services. 


Figure  6  -  Wide  Area  Motion  Imagery  (WAMI)  data. 


Given  WAMI  [53,  54]  data,  shown  in  Fig.  6,  we  seek  the 
benefits  that  are  enabled  from  an  enterprise  network.  For 
example,  when  the  user  designates  an  area  of  interest,  the 
machine  can  then  detect  and  track  targets  (LI  fusion)  and 
access  contextual  information  through  data  mining  (L2/L3 
fusion)  to  enhance  understanding  (e.g.,  target  type, 
identification  (ID)  and  activity).  Finally,  the  results  are  used  to 
query  the  sensors  to  get  more  information  (L4  fusion),  store 
the  results,  present  to  the  user  (L5  fusion)  and  disseminate 
back  to  the  cloud  for  mission  awareness  (L6  fusion). 

B.  Information  Management  Contextual  Tracking 

In  scenario  1,  raw  (images,  text),  filtered  (tracks,  keywords), 
and  fused  (tracks  with  classification  and  ID  labels)  are  sent  to 
user  stations.  For  the  five  cases  (1-5),  we  model  data  scaled  to 
the  relative  volume  values.  We  assume  that  there  five 
distributed  operators  are  maintaining  surveillance.  The 
complexity  from  Case  1  to  Case  5  is  akin  to  tracking  a  few 
targets  (8  =  40/5)  to  that  of  trying  to  maintain  tracks  over  the 
entire  context  of  a  city  (200  tracks  =  1000/5).  For  users,  they 
need  information  fusion  support  to  track  these  many  targets. 
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Figure  7  -  Traditional  JDL  Data  Fusion 


Assuming  raw  data  messaging,  it  is  intuitive  that  the  filtered 
information  can  reduce  the  volume  by  ruling  out  unnecessary 
data  (see  Fig.  7).  The  results  indicate  that  from  the  traditional 
JDL  methods,  there  is  little  to  no  information  management 
from  the  user,  mission,  or  machine  analytics. 

In  the  second  notional  scenario,  we  are  interested  in  seeing  the 
benefits  of  user  involvement,  machine  analytics,  and 
information  management  to  support  distributed  users.  Not  all 
users  require  the  same  collection  information;  however  they 
can  query  the  information  they  need  from  the  cloud  to  be 
presented  on  their  displays.  In  this  case,  we  use  a 
publish/subscribe  (Pub/Sub)  architecture  to  afford  the 
interactions  of  distributed  users. 

Using  Pub/Sub  information  management,  it  is  important  to 
note  that  over  a  different  set  of  users  desiring  situational 
context,  with  two  data  feeds  (video  to  tracks  and  text  to 
keywords)  and  40  to  200  tracks,  there  is  a  break  point  in 
operations.  For  this  case,  if  you  try  and  normalize  over  the 
time  duration  of  the  scenario,  the  fused  information  accounts 
to  three  pieces  of  information  for  each  image.  For  the  case  of 
two  operators,  they  can  follow  more  objects  by  specializing  in 
the  data  source  and  area  of  designated  interest  to  highlight 
tracks.  The  interesting  part  is  the  difference  between  5  to  10 
operators  in  that  there  is  an  order  of  magnitude  more 
information  being  passed  around  as  many  users  (assumed)  are 
requesting  the  same  information. 

The  notional  Scenario  2  speaks  to  team  management.  With  too 
many  operators  looking  at  similar  data,  there  could  be  overlap 
in  functions.  Using  contextual  information  and  user-defined 
operating  pictures  (UDOP)  displays,  information  can  deliver 
machine  analytic  results  to  the  appropriate  user.  Each  UDOP 
provides  a  tailored  situational  display  and  the  user  can  call  the 
raw,  filtered,  or  fused  data.  For  this  scenario,  we  compare  the 
filtered  and  fused  data  as  the  user  does  not  want  the  raw  data. 


Figure  8  -  Data  Fusion  with  info  exploitation. 


Fig.  8  demonstrates  possible  needs  to  extend  the  JDL  model. 
In  this  case,  using  information  management  sends  the  correct 
data  to  the  correct  user  (by  request)  for  user  assessment. 
However,  using  the  combination  of  information  fusion  and 
contextual  awareness  in  a  pub/sub  enterprise,  only  the  fused 
data  is  going  back  and  forth  -  saving  volumes  of  data 
transactions.  Here,  we  assume  such  things  as  visualization 
(versus  the  entire  filtered  image  and  text),  resident  overhead 
imagery  to  overlay  tracks  and  text  reports  at  the  distributed 
site,  and  command  and  control  interfaces  that  allow  each  user 
to  know  the  updates  from  the  other  users. 


- Raw 

- Filtered 
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VI.  Conclusions 

Since  the  inception  of  the  JDL  model,  it  has  served  as  a 
process  model  to  explain  information  fusion.  This  paper 
revisits  the  JDL  based  on  technology  developments  in  the  last 
decade  as  discussed  in  the  ISIF  community.  We  highlight 
three  information  exploitation  technology  developments 
affecting  the  JDL  model  that  include:  (1)  an  enterprise 
architecture  that  supports  information  management  to  store 
and  access  data  through  cloud  computing,  (2)  machine 
analytics  for  data  mining,  and  (3)  man-machine  interfaces  to 
support  users  as  active  observers  of  contextual  reasoning  (L5). 
Acknowledgement:  We  appreciate  the  reviewers’  insightful  additions. 
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